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EXECUTIVE  SUMMARY 


In  the  last  few  years,  the  Electronic  Chart  Display  and  Information  System  (ECDIS)  has 
emerged  as  a  powerful  addition  to  the  modem  bridge,  offering  the  possibility  of  effecting 
major  changes  in  the  navigation  process  while  improving  the  safety  and  efficiency  of  maritime 
operations.  By  superimposing  electronic  chart,  ship's  position,  and  RADAR  on  one  display, 
ECDIS  has  the  potential  to  improve  the  accuracy  of  navigation,  increase  awareness  of 
dangerous  conditions,  and  reduce  the  mariner's  workload.  An  ECDIS  is  defined  by  the 
International  Maritime  Organization  (IMO)  as  a  system  which  is  compliant  with  the  IMO 
Performance  Standards  for  ECDIS.  Systems  which  are  not  “IMO  compliant”  are  categorized 
as  Electronic  Chart  Systems  (ECS). 

There  is  a  worldwide  effort  to  study  this  new  and  promising  technology.  The  International 
Maritime  Organization  and  the  International  Hydrographic  Organization  (IMO/IHO) 
Harmonization  Group  on  ECDIS  (HGE)  began  the  process  of  developing  a  draft  performance 
standard  for  ECDIS  (IMO/IHO,  1989  &  1992).  This  performance  standard  was  completed 
and  adopted  by  IMO  under  Resolution  A817  in  November  1995.  This  report  describes  the 
United  States  Coast  Guard's  (USCG)  four  year  effort  to  examine  potential  effects  of  ECDIS 
both  at-sea  and  in  a  full-mission  ship's  bridge  simulator.  The  results  and  insights  garnered 
from  these  studies  contributed  to  U.S.A.  position  reports  on  the  IMO  Performance  Standard 
for  ECDIS.  These  various  position  reports  were  provided  to  the  IMO  Safety  of  Navigation 
Subcommittee  from  1991  to  1995. 

The  USCG  developed  a  program  to  thoroughly  test  the  adequacy  of  the  Performance  Standards 
for  ECDIS.  This  program  consisted  of  four  sea  trials  and  one  man-in-the-loop  simulator 
experiment  (Table  ES.l  summarizes  the  experiments  for  the  U.S.  Test  and  Evaluation 
Program).  This  report  discusses  the  background,  experimental  design,  instrumentation,  results 
and  conclusions  of  the  following  experiments. 

OCTOBER  1990  INITIAL  SEA  TRIALS  -  Confined  waterway/harbor  navigation  tests 
conducted  on  board  the  U.S.  Merchant  Marine  Academy  T/V  KINGS  POINTER.  Mariner 
performance  was  evaluated  in  terms  of  deviation  from  intended  track  (XTD)(s'cc  table  ES.l). 
The  subjects  were  one  master  and  one  cadet  on  an  established  route  with  predetermined  mrns 
and  waypoints.  Deviation  from  intended  track  (cross  track  distance  or  XTD)  was  monitored. 
Results  indicated  that  overall  track  keeping  performance  significantly  improved  when 
navigating  with  ECDIS. 

OCTOBER  1991  KINGS  POINTER  SEA  TRIALS  -  Confined  waterway/harbor  navigation 
tests  conducted  on  board  the  U.S.  Merchant  Marine  Academy  T/V  KINGS  POINTER  on  the 
East  River  in  New  York  (sec  table  ES.l).  Building  on  1990  Sea  Trials,  Mariner  performance 
was  again  evaluated  in  terms  of  deviation  from  intended  track  (XTD).  The  subject  pool 
consisted  of  20  Mariners:  4  pilots,  5  masters,  4  mates  and  7  cadets.  Results  of  these  trials 


ES- 1 


indicated  that  overall  navigational  performance  of  mariners  is  significantly  enhanced  when 
using  ECDIS.  Additionally,  no  significant  differences  were  found  in  navigational  perfor¬ 
mance  between  varying  experience  levels  of  mariners  while  navigating  with  ECDIS  . 

SEPTEMBER  -  OCTOBER  1992  SIMULATOR  EXPERIMENTS  -  A  series  of  simulator 
experiments  were  conducted  at  Marine  Safety  Intemational/Computer  Aided  Operations 
Research  Facility  (MSI/C AORF).  Experimental  scenarios  consisted  of  port  arrivals  and 
departures  with  one  mariner  alone  on  the  bridge  responsible  for  navigation,  collision 
avoidance,  and  bridge  management  tasks.  The  simulator  provided  an  environment  to  test 
mariners  under  stress,  without  the  danger  of  an  accident.  The  subject  pool  consisted  of  six 
expert  mariners,  four  masters  and  two  mates.  Test  results  indicated  that  ECDIS  increased 
navigation  safety  both  by  improving  the  accuracy  of  ship  tracks  and  by  allowing  the  mariners 
to  spend  a  greater  proportion  of  time  on  collision  avoidance.  With  automatic  updating  of 
position,  ECDIS  decreased  the  workload  for  navigation.  Without  automatic  updating  of 
position,  ECDIS  became  ineffective.  Mariners  expressed  a  preference  for  a  relatively  simple 
chart  display  for  route  monitoring,  with  the  immediate  availability  of  a  larger  set  of  features. 
No  measurable  effects  of  RADAR  features  on  ECDIS  were  found,  although  mariners  felt  that 
this  should  be  a  valuable  addition. 


TABLE  ES.l  -  SUMMARY  OF  U.S.  EXPERIMENTS 


DATE 

TYPE 

1  MARINERS 

1  ECDIS  * 

October  1990 

2  days 

Sea  Trials 

T/V  KINGS  POINTER 

1  Master 

1  Cadet 

1)  NavGraphic  II: 

Trimble  Navigation 

October  1991 

8  days 

Sea  Trials 

T/V  KINGS  POINTER 

4  Pilots 

5  Masters 

4  Mates 

7  Cadets 

1)  PINS  9000:  OSL 

Sep  -  Oct  1992 

30  days 

Ashore  Simulator 
MSI/CAORF 

4  Masters 

2  Mates 

1) PINSVME:  OSL 

2)  Disk  Navigation  Sys:  Robertson 

Jan  -  Feb  1993 

18  days 

Sea  Trials 

USCG  Cutter 
BITTERSWEET 

7  Qualified 

Officers  of  the 

Deck  (OOD) 

1)  ECPINS:  OSL 

March  -  April  1993 

10  days 

Sea  Trials 

M/V  KINGS  POINTER 

4  Pilots 

4  Masters 

5  Mates 

1)  ECPINS  OSL 

2)  Hydraut  EC  Series  R&H 

3)  ATLAS  9800  Atlas  Electronik 

*  Note:  Any  mention  of  a  manufacturer  or  product  in  this  report  is  not  intended  as  an 

endorsement  by  the  U.S.  Coast  Guard,  but  rather  as  an  acknowledgement  that  the 
system  cited  was  used  in  conjunction  with  the  U.S.  ECDIS  Test  and  Evaluation 
Program. 
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JANUARY  -  APRIL  1993  BITTERSWEET  &  KINGS  POINTER  SEA  TRIALS  -  Two  sets  of 
sea  trials  were  conducted  back  to  back.  The  first  trials  were  designed  to  closely  monitor 
mariners  as  they  used  ECDIS  during  normal  at-sea  conditions.  The  trials  consisted  of  harbor 
and  coastal  navigation  trials  during  January  and  February  1993  aboard  the  USCG  Cutter 
BITTERSWEET,  a  180'  (55m)  ocean-going  buoy  tender.  All  of  the  ship's  seven  qualified 
officers  of  the  deck  (OODs)  participated  as  test  subjects.  Three  of  the  OODs  were  senior 
watchstanders  and  four  were  junior  watchstanders.  Extensive  ship  control  data  were  recorded, 
and  a  variety  of  human  factors  techniques  were  used  to  measure  performance,  workload  and 
operator  reactions.  The  second  sea  trials  were  conducted  aboard  the  new  U.S.  Merchant 
Marine  Academy  training  vessel,  M/V  KINGS  POINTER.  Thirteen  experienced  mariners 
participated  in  the  trials  in  the  upper  East  River/Manhasset  Bay,  New  York  area:  four  pilots, 
four  masters  and  five  mates. 

Following  the  underway  period,  mariners  completed  a  written  questionnaire  and  then  were 
verbally  debriefed.  Results  of  the  debriefing  as  well  as  other  data  indicated  that  navigation 
workload  decreased  when  using  ECDIS.  The  mariners  felt  that  the  ECDIS  display  must  be 
simple  and  uncluttered,  but  all  other  relevant  navigational  information  must  be  readily 
available.  Mariners  felt  ECDIS  made  an  excellent  contribution  to  safe  navigation  and  that  a 
fiilly  capable,  fully  integrated  RADAR/ARPA  is  necessary  for  navigation  and  collision 
avoidance,  and  poses  no  loss  in  capability.  Mariners  indicated  that  since  all  three  ECDIS 
devices  had  very  different  user-interfaces,  a  minimum  number  of  standard  functions  should  be 
designed  into  ECDIS  specifications  to  help  prevent  operator  errors  due  to  unfamiliar 
interfaces. 

Each  test  was  designed  to  incorporate  known  methodology  and  advanced  technology  to 
thoroughly  evaluate  the  Draft  Performance  Standard  for  ECDIS.  As  the  experiments 
progressed,  the  technology  became  more  sophisticated,  with  information  being  sought  in  more 
breadth  and  depth.  Each  experiment,  building  on  the  findings  of  the  previous,  was  designed  to 
ask  more  indepth  questions,  to  gain  more  complete  information.  The  result  of  this  was  one  of 
the  most  comprehensive  studies  done  on  ECDIS  performance. 

The  USCG  IMO/IHO  HGE  representative  for  ECDIS  standards  development  used  the  findings 
from  the  October  1991  Sea  Trials  to  develop  a  position  paper  on  several  of  the  ECDIS 
Provisional  Performance  Standard  Issues.  This  position  paper  along  with  other  member 
nations  recommendations  were  combined  to  create  an  IHO/IMO  HGE  Draft  Performance 
Standard  for  ECDIS  (IHO/IMO,  Sep  1992).  Insight  and  recommendations  based  on  the  later 
experiments  were  used  to  improve  the  IMO  Draft  Performance  Standard  for  ECDIS,  which 
became  MSC/Cir.  637(IMO,  May  ‘94). 

There  are  several  key  findings  produced  by  this  body  of  work.  It  has  shown  consistently  that 
ECDIS  can  provide  equivalent  or  greater  safety  than  that  provided  by  the  use  of  paper  chart 
and  more  traditional  methods  of  navigation.  Another  key  finding  is  that  navigation  workload 
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is  reduced,  allowing  the  mariner  to  concentrate  on  collision  avoidance  and/or  other  tasks  of 
similar  importance.  In  the  area  of  user-interface  design,  it  was  found  that  the  mariner  agreed 
for  the  most  part  with  the  Draft  Performance  Standard  for  ECDIS.  They  wanted  an 
"uncluttered"  display  during  route  monitoring,  with  more  features  immediately  available  if 
needed.  The  findings,  however,  did  suggest  several  places  where  the  Draft  Performance 
Standard  for  ECDIS  were  lacking.  These  findings  were  also  reported  to  the  IMO/IHO 
Harmonization  Group  on  ECDIS,  and  precipitated  more  changes  to  the  Performance  Standards 
Document,  up  until  its  adoption  by  IMO  (Res.  A817)  in  November  1995  (IMO,  Nov  ‘95). 
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I.O  TLS.  ECDTS  TEST  AND  EVALUATION  PROGRAM 


1.1  INTRODUCTION 

In  1990,  the  United  States  Coast  Guard  (USCG)  was  tasked  with  determining  the  potential  of 
Electronic  Chart  Display  and  Information  System  (ECDIS)  to  improve  safety  of  navigation. 
Provisional  Performance  Standards  for  ECDIS  had  recently  been  published  by  the  International 
Maritime  Organization  (IMO)  in  1989  (IMO,  1989).  This  standard  was  established  to 
facilitate  the  worldwide  development  and  utilization  of  ECDIS  as  a  shipboard  navigation 
system.  These  Provisional  Performance  Standards  became  a  guide  for  the  USCG  to  develop  a 
comprehensive  test  and  evaluation  program.  The  goals  of  the  program  were  to: 

•  determine  the  capabilities  and  limitations  of  current  and  prototype  ECDIS 
devices  for  improving  safety  of  navigation 
.  evaluate  the  adequacy  of  proposed  ECDIS  design  and  performance  standards 
.  ensure  that  human  factors  considerations  are  incorporated  into  the  design, 
operation,  and  performance  of  ECDIS 

The  USCG  is  involved  in  a  variety  of  national  and  international  ECDIS  related  projects 
designed  to  meet  the  goals  of  the  USCG  test  and  evaluation  program  (Alexander  and  Black, 
1993).  The  tests  listed  below  were  specifically  designed  and  carried  out  by  the  USCG 
Research  and  Development  Center  (R&DC)  to  assist  in  meeting  these  ECDIS  program  goals. 

.  OCTOBER  1990  INITIAL  SEA  TRIALS 
.  OCTOBER  1991  KINGS  POINTER  SEA  TRIALS 
.  SEPTEMBER  -  OCTOBER  1992  SIMULATOR  EXPERIMENTS 
.  JANUARY  -  APRIL  1993  BITTERSWEET  &  KINGS  POINTER  SEA  TRIALS 

The  intention  of  this  report  is  to  provide  a  summary  of  the  objectives,  experimental  design, 
instrumentation,  and  results  for  the  above  four  tests.  A  more  thorough  description  of  the 
October  1991  KINGS  POINTER  Sea  Trials  can  be  found  in  Gonin,  et.  al.  1992.  The 
September-October  1992  Simulator  Experiment  is  described  in  much  more  detail  in  Smith,  et. 
al.  1995  and  also  in  the  paper  Gonin,  et.  al.  1994.  More  detailed  information  on  the  final  test, 
the  January-April  1993  BITTERSWEET  &  KINGS  POINTER  Sea  Trials,  can  be  found  in 
Gonin,  et.  al.  ‘95. 

1.2  BACKGROUND 

Due  to  significant  technological  advances  in  radionavigation,  computer  hardware  and  software, 
digital  chart  data,  sensor  integration,  and  geographical  information  system  (GIS)  technologies, 
great  improvements  in  navigation  are  made  possible.  ECDIS  combines  these  technologies  and 
data  to  become  a  real-time  GIS;  this  implies  that  ECDIS  is  a  system  capable  of  continuously 
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determining  a  vessel's  position  in  relation  to  land,  charted  objects,  aids  to  navigation,  and 
unseen  hazards.  ECDIS  has  the  potential  to  improve  navigation  accuracy,  improve  mariners' 
awareness  of  potentially  dangerous  situations,  and  reduce  the  workload  on  the  bridge. 

In  the  United  States,  three  government  agencies  are  responsible  for  providing  essential  services 
for  the  use  of  ECDIS. 

.  The  U.S.  Department  of  Defense  is  providing  the  Navstar  Global  Positioning 
System  (GPS)  for  position  input  into  ECDIS. 

•  The  U.  S.  Coast  Guard  has  installed  a  Differential  GPS  (DGPS)  broadcast 

service  which  provides  more  accurate  position  information  than  GPS  alone  in 
harbor  navigation  areas. 

.  The  Coast  Survey  (CS)  Branch  of  the  National  Oceanic  and  Atmospheric 

Administration  (NOAA)  is  expected  to  provide  official  digital  chart  data  for 
ECDIS. 

With  these  services,  mariners  will  increasingly  rely  on  automated  radionavigation  positioning 
and  electronic  chart  displays.  These  new  technologies  will  give  mariners  accurate,  timely,  and 
dependable  data.  The  challenge  lies  in  developing  standards  which  will  provide  a  clear  and 
easy  to  understand  display,  on  a  system  which  is  user-friendly. 

In  order  to  determine  the  effects  of  these  new  technologies  on  navigation  performance,  the 
USCG  has  had  an  active  human  factors  research  program  for  more  than  a  decade.  The 
program  focused  on  the  effectiveness  of  ECDIS-style  integrated  displays  for  the  specific 
operation  of  harbor/harbor  approach  piloting,  especially  in  reduced  visibility.  Systems  have 
been  examined  both  at  sea  (Cooper  and  Bertsche,  1981;  Roeber,  1981)  and  on  real-time-man- 
in-the-loop  simulators  (Gynther  and  Smith,  1988:  Smith  and  Mandler,  1992).  These  reports 
describe  important  findings  on  the  effectiveness  of  display  features,  the  influence  of 
positioning  error  on  the  effectiveness  of  these  displays,  and  the  influences  of  a  RADAR 
overlay  on  piloting  performance.  These  reports  were  the  base  from  which  the  current  effort 
was  launched. 

There  are  generally  considered  to  be  two  categories  for  electronic  chart-based  navigation 
systems:  ECDIS  and  Electronic  Chart  Systems  (ECS).  ECDIS  is  defined  as  a  system  which  is 
compliant  with  the  International  Maritime  Organization  (IMO)  Performance  Standards  for 
ECDIS,  systems  which  are  not  “IMO  compliant”  are  categorized  as  ECS.  As  the  technology 
continues  to  improve  and  standards  continue  to  be  developed,  rnanufacmrers  continually  strive 
for  systems  which  are  IMO  compliant  (i.e.,  ECDIS).  During  the  time  period  of  each  of  the 
studies  described  in  this  report,  the  performance  standards  were  in  flux,  and  few 
rnanufacmrers  could  keep  up  with  the  changes.  Therefore  the  “ECDIS”  employed  in  these 
smdies  were  of  the  highest  level  of  sophistication  available  at  the  time,  yet,  were  only  able  to 
meet  some  of  the  IMO  requirements. 
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2.0  OCTOBER  1990  INITIAL  SEA  TRIALS 


Objective 

The  objective  of  the  initial  sea  trials  was  to  determine  the  minimal  requirements  for  an  ECDIS 
device  that  would  increase  navigation  safety.  The  initial  sea  trial  experiment  was  designed  to 
develop  the  expertise  and  methodology  needed  for  testing  ECDIS.  Commercially  available 
ECDIS  technology  was  evolving  so  rapidly  that  a  prototype  system  designed  specifically  for 
these  ECDIS  evaluations  could  become  obsolete  too  quickly.  It  became  the  goal  of  the  USCG 
R&DC  to  concentrate  on  the  design  of  an  experiment  which  could  be  performed  with  any  off- 
the-self  ECDIS  so  that  technology  would  not  limit  the  evaluation. 

2.1  EXPERIMENTAL  DESIGN 

Subjects 

The  pool  of  subjects  consisted  of  one  master  and  one  cadet. 

Method 

A  series  of  transits  were  made  by  one  master  and  one  cadet  on  an  established  route  with 
predetermined  turns  and  waypoints  in  a  fairly  confined  waterway.  The  track  was  basically  a  zig¬ 
zag  in  the  channel.  Although  a  somewhat  artificial  route,  it  was  intended  to  compress  many  turns 
in  a  short  distance,  so  that  differences  in  navigation  performance  could  easily  be  seen.  Deviation 
from  intended  track  was  monitored  and  recorded  for  the  three  types  of  navigation  runs  as 
illustrated  in  Table  2.1.1.  These  methods  of  navigation,  with  slight  variations,  were  used  in  all 
the  experiments  performed  with  ECDIS.  Traditional  navigation  and  RADAR  navigation  were 
the  baseline  methods  used  to  compare  ECDIS  navigation  performance.  Deviation  from  intended 
track  was  used  to  determine  mariner  performance  (Table  2.1 .2). 


Table  2.1.1  Experimental  Design  Factors 


Factors  Controlled 

Levels 

1)  Navigation  Type 

Traditional 

RADAR 

ECDIS 

Table  2.1.2  Experimental  Design  Performance  Measures 


Objective  Measures 

1)  cross  track  distance 

Subjective  Measures 

None 
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2.2  INSTRUMENTATION 


These  harbor  navigation  tests  were  conducted  onboard  the  United  States  Merchant  Marine 
Academy  (USMMA)  T/V  KINGS  POINTER,  a  150-foot  (45-meter)  ocean  going  tug,  using  the 
NavGraphic  II  -  Electronic  Chart  System  (ECS)  from  Trimble  Navigation,  USA  (Table  2.2.1). 
At  the  time,  this  was  one  of  the  more  capable  ECSs  commercially  available. 

Table  2.2.1  Instrumentation  


Experimental  Platform 

T/V  KINGS  POINTER,  a  150-foot  (45-meter)  ocean  going  tug 

ECS  Platform 

NavGraphic  II  -  Electronic  Chart  System  (ECS),  from  Trimble  Navigation,  USA 

Features: 

raster  scan  digital  chart  data  using  a  CD  ROM  for  storage  and  access 

integrated  GPS  (primary)  and  LORAN-C  (secondary)  position  sensors 

capable  of  waypoint  navigation 

displays  for  course  over  grotmd  (COG),  speed  over  ground  (SOG),  cross  track 
distance  (XTD),  latimde  and  longimde  (lat/lon),  and  range  and  bearing 

capability  of  setting  alarms  (i.e.  entering  a  mariner-determined  danger  area) 

Comment 

awkward  user  interface;  difficult  and  non-inmitive  navigation  menu 

2.3  RESULTS 

Cross  track  data  were  analyzed  and  the  results  indicated  that  overall  track  keeping  performance 
is  improved  while  navigating  with  ECDIS.  Both  the  cadet  and  master  felt  they  had  a  better  ' 
feeling  for  where  they  were  in  the  route.  They  seemed  to  be  able  to  anticipate  turns  more 
easily  when  using  the  ECDIS.  At  one  point  the  cadet  said,  "Using  the  ECDIS  is  like  playing  a 
video  game,  just  keep  the  vessel  icon  on  the  track  line. " 

The  results  from  this  initial  trial  were  encouraging,  although  there  was  still  much  to  be  done  in 
determining  the  potential  of  ECDIS.  In  this  trial  only  two  subjects  were  used  and  they  each 
performed  the  same  run  many  times.  As  often  happens  in  such  a  case,  the  later  runs  showed 
better  results  than  the  earlier  runs;  this  is  commonly  referred  to  as  "the  learning  effect. 
Additionally,  the  use  of  predefined  turns  and  waypoints  were  an  artificial  constraint  on  the 
mariner.  Solving  some  of  these  problems  and  improving  upon  the  test  methodology  were  part 
of  the  goals  for  the  follow-on  sea  trials. 
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^.n  OCTOBER  1991  KINGS  POINTER  SEA  TRIALS 

Objective 

With  the  completion  of  the  initial  sea  trials,  a  solid  foundation  for  testing  and  evaluating 
ECDIS  was  established.  The  objective  was  to  conduct  a  more  comprehensive  set  of  trials  in 
this  follow-on  experiment.  This  next  experiment  tested  ECDIS  performance  standards,  and  the 
effect  of  ECDIS  on  navigation  safety  and  mariner  workload.  Additionally,  the  state-of-the-art 
in  ECDIS  technology  had  moved  ahead.  Therefore,  a  more  capable  ECDIS  device  was  used 
for  the  evaluation.  This  device  was  able  to  support  configuration  changes  (i.e.  graphics  and 
alphanumerics-size,  color,  type  and  quantity),  integration  of  a  variety  of  sensors  (i.e.  GPS, 

DGPS,  LORAN,  speed,  heading,  depth,  RADAR,  etc.),  and  use  vector  based  digital  chart  data 
(i.e.  database  of  chart  objects  and  attributes  displayed  on  a  screen). 

3.1  EXPERIMENTAL  DESIGN 

Subjects 

The  subject  pool  consisted  of  20  mariners:  4  pilots,  5  masters,  4  mates  and  7  cadets. 

Method 

In  order  to  help  correct  for  the  learning  effects  experienced  in  our  initial  trials,  a  larger  pool  of 
subjects  and  a  more  robust  experimental  design  were  used.  Additionally,  to  make  the 
experiment  as  realistic  as  possible,  we  had  each  mariner  plot  his/her  own  route  around  Rikers 
Island,  NY. 

The  United  States  Merchant  Marine  Academy's  ocean  going  tug,  T/V  KINGS  POINTER,  was 
used  for  the  trials.  This  vessel  and  its  location  (i.e.  near  the  East  River  in  New  York  City  in 
the  vicinity  of  Rikers  Island  with  its  abundance  of  navigation  aids  such  as  buoys,  lights, 
stacks,  etc.,  and  a  sharp  turn  in  the  waterway)  provide  an  ideal  test  platform/location  to 
conduct  navigation  tests  and  evaluations  in  confined  waterways. 

A  test  matrix  was  developed  which  attempted  to  reduce  some  of  the  problems  in  the  initial 
trials,  as  well  as  build  on  the  areas  that  produced  good  results.  Once  again,  cross  track 
distance  was  used  to  measure  performance.  In  this  experiment,  however,  the  use  of  an  exit 
survey  was  added  (Table  3.1.1). 


Table  3.1.1  Experimental  Design  Performance  Measures 


Objective  Measures 

1)  Cross  Track  Distance 

Subjective  Measures 

1)  Exit  Survey 

5 


The  experiment  evaluated  the  performance  of  four  different  types  of  mariners,  while  executing 
three  types  of  navigation  runs,  during  day  and  night  time  transits,  in  west/south  and  north/east 
directions;  during  periods  of  good  and  poor  visibility,  in  a  confined  waterway,  using  mariner 
developed  routes.  These  Experimental  design  factors  are  illustrated  in  Table  3.1.2  and  3.1.3. 
Poor  visibility  was  simulated  by  covering  the  bridge  windows  with  opaque  paper. 


Table  3.1.2  Experimental  Design  Factors 


Factors  Controlled 

Levels 

1)  Navigation  Type 

Traditional 

RADAR 

ECDIS 

2)  Time  of  Day 

Day 

Night 

3)  Direction 

West  and  South 

North  and  East 

4)  Visibility 

Poor 

Good 

Table  3.1.3  Test  Matrix  and  Runs  Performed 

■  II  -  -  -""ii' - ^ 


Run 

No. 

Direction 

Navigation 

Method 

Condition 

Visibility 

Condition 

Time 

Runs 

Performed 

1 

A-B 

TRADITIONAL 

Good 

Day 

5 

2 

West/South 

Night 

2 

3 

RADAR 

Poor 

Day 

2 

4 

Night 

2 

5 

ECDIS 

Good 

Day 

4 

6 

Night 

0 

7 

Poor 

Day 

5 

8 

Night 

2 

9 

B-C 

TRADITIONAL 

Good 

Day 

6 

10 

North/East 

Night 

2 

11 

RADAR 

Poor 

Day 

2 

12 

Night 

1 

13 

ECDIS 

Good 

Day 

3 

14 

Night 

0 

15 

Poor 

Day 

6 

16 

Night 

2 
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A  secondary  goal  of  this  sea  trial  was  to  evaluate  some  of  the  ECDIS  functions  and  features 
proposed  in  the  IMO  Provisional  Performance  Standard  for  ECDIS  (IMO,  1989).  This  task  was 
accomplished  by  having  each  mariner  fill  out  an  evaluation  after  using  the  ECDIS.  A 
compilation  of  the  responses  received  from  all  experiments,  along  with  other  results  and 
conclusions  would  be  combined  and  used  to  develop  the  USCG  response  to  IMO  on  the 
Provisional  Performance  Standard  for  ECDIS. 

In  order  to  ensure  that  there  was  a  specific  distinction  between  traditional  and  RADAR 
navigation  for  determining  mariner  performance,  good  and  poor  visibility  was  assigned  to  each 
respectively.  Typically,  a  mariner  uses  traditional  or  visual  piloting  as  a  primary  method  of 
navigation  in  good  visibility  and  uses  the  RADAR  quite  heavily  in  restricted  visibility. 
Performing  traditional  navigation  without  the  RADAR  in  restricted  visibility  is  not  feasible  and 
performing  RADAR  navigation  in  good  visibility  is  quite  similar  to  traditional  or  visual  piloting 

3.2  INSTRUMENTATION 

The  Precise  Integrated  Navigation  System  (PINS)  9000  was  used  as  the  ECDIS  platform.  This 
system  uses  vector  chart  data  produced  by  its  manufacturer.  Offshore  Systems  Ltd  (OSL), 
Canada.  Features  of  the  PINS  9000  are  described  in  Table  3.2.1.  The  use  of  vector-based  data. 


Table  3.2.1  Instrumentation 


Experimental  Platform 

TA^  KINGS  POINTER,  a  150-foot  (45-meter)  ocean  going  tug 

ECDIS  Platform 

Precise  Integrated  Navigation  System  (PINS)  9000,  manufactured  by  Offshore  Systems 
Ltd  (OSL),  Canada 

Features: 

vector-based  digital  charts  data  produced  by  its  manufacturer,  which  allows  for  fast 
chart  refresh  and  extensive  "zooming"  capabilities 

integration  of  ship's  Gyro  Compass  and  Differential  Global  Positioning  System  (DGPS) 

capable  of  waypoint  navigation 

displays  for  course  over  ground  (COG),  speed  over  ground  (SOG),  cross  track  distance 
(XTD),  latitude  and  longitude  (lat/lon),  and  range  and  bearing 

capability  of  setting  alarms  (i.e.  entering  a  mariner-determined  danger  area) 

two  independent  monitors;  one  on  the  bridge  for  navigation,  the  other  in  the  chart  room 
for  monitoring 

Comment 

user  interface  advanced,  allows  for  custom  screens,  and  many  sensor  displays; 
navigation  menu  more  user-friendly,  but  still  non-intuitive. 

allows  for  fast  chart  refresh  and  extensive  "zooming"  capabilities.  This  system  also  allows 
custom  screens  and  many  sensor  inputs.  This  flexibility  makes  it  an  ideal  tool  for  test  and 
evaluation.  Several  screens  were  created  with  different  alphanumeric  and  chart  information  taken 
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from  the  proposed  IMO  Provisional  Performance  Standard  for  ECDIS  (IMO,  1989).  This  gave 
the  mariner  an  opportunity  to  select  a  specific  screen  and  gave  us  good  feedback  on  the  types  of 
information  mariners  felt  they  needed  on  an  ECDIS. 

3.3  RESULTS 

Two  different  approaches  were  used  to  analyze  the  data  collected.  The  first  approach  analyzed 
the  different  skill  levels  and  navigation  methods  (which  included  good  and  poor  visibility)  using 
all  the  cross  track  distance  data.  This  approach  created  a  cumulative  frequency  distribution  curve 
for  each  set  of  cross  track  distance  data.  This  provided  a  presentation  of  the  proportion  of 
observations  that  are  less  than  or  equal  to  the  upper  limit  of  the  data  set  selected.  Other 
descriptive  statistics  such  as  summaries  of  average  (mean),  standard  deviation,  minimum  and 
maximum  values  were  also  calculated. 

Next,  after  correcting  for  the  high  level  of  autocorrelation  for  the  cross  track  distance  data, 
Analysis  of  Variance  (ANOVA)  was  performed  on  the  remaining  variables  (west/south/north/ 
east  direction  and  day/night  transits). 

Table  3.3.1  was  created  to  summarize  some  specific  values  of  cross  track  distance  received  from 
the  Cumulative  Distribution  Function  (CDF)  of  four  categories  of  runs.  These  are  all  combined 
runs,  all  ECDIS  runs,  all  traditional  runs,  and  all  RADAR  runs.  It  can  be  seen  from  this  table 
that  the  runs  with  ECDIS  show  by  far  the  lowest  cross  track  distance  of  all  of  the  methods  tested. 


Table  3.3.1  Summary  of  Cumulative  Distribution  Frequencies  of  Cross  Track  Distance 
_ (deviation  from  intended  track  in  meters)  of  50%  and_95%^oints _ 


ALL  RUNS 

ECDIS 

TRADITIONAL 

RADAR  ONLY 

50% 

95% 

50% 

95% 

50% 

95% 

50% 

95% 

ALL  SUBJECTS 

15 

76 

10 

44 

26 

86 

22 

114 

PILOTS 

15 

91 

10 

46 

37 

95 

20 

158 

MASTERS 

15 

91 

8 

46 

26 

107 

20 

93 

MATES 

17 

58 

12 

42 

25 

57 

32 

90 

CADETS 

18 

78 

12 

42 

25 

80 

NOT  TESTED 

3.3.1  Navigation  Methods  Compared 

Mariner  performance  for  each  of  the  three  types  of  navigation  methods  tested  (i.  e.  ECDIS, 
RADAR,  and  traditional  piloting)  are  plotted  in  Figure  1.  There  is  a  significant  difference 
between  mariners  performing  ECDIS  navigation  versus  more  traditional  methods,  mean 
(average)  difference  of  almost  20  meters. 

When  using  ECDIS,  no  significant  difference  in  performance  between  the  four  experience  levels 
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FIGURE  2  -  ECDIS  RUNS 


of  mariners  was  detected(see 
Figure  2).  The  difference  in 
mean  cross  track  distance 
between  all  four  experience 
levels  was  4.4  meters.  With 
traditional  navigation  the 
difference  between  mean  cross 
track  distance  was  12.6 
meters  and  with  RADAR 
only  navigation  the  difference 
was  12.1  meters. 

Figure  3  shows  CDF  of  cross 
track  distance  of  all 
experience  levels  performing 
traditional  navigation.  In  this 
analysis  the  mates  performed 
slightly  better  than  the  other 
three  experience  levels. 

After  discussion  with 
mariners  and  scientists,  the 
general  consensus  was  that 
mates  were  the  most 
conscientious  of  all  four 
groups  because  of  their 
combination  of  recent 
training  and  experience. 

Although  helm  commands 
were  not  part  of  the  test,  it 
was  observed  that  the  mates 
gave  many  helm  commands 
to  maintain  track.  Pilots,  and 
to  a  lesser  degree  masters, 
knew  that  they  were  off 
track,  but  felt  comfortable 
with  their  position  when  they 
were  navigating  by 
traditional  methods.  They 
felt  it  unnecessary  to  give  a 
lot  of  helm  commands  to  get 
back  perfectly  on  track. 
Cadets,  on  the  other  hand, 
have  very  little  experience, 
but  a  lot  of  training.  Because 
of  this  inexperience,  they 
were  less  able  to  anticipate 
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FIGURE  4  -  RADAR  RUNS 


their  new  position.  This  resulted 
in  many  helm  commands  given 
in  an  effort  to  stay  on  intended 
track. 

Figure  4  shows  RADAR  runs  by 
mariner  type.  Cadets  did  not 
participate  in  this  portion,  as  they 
were  not  considered  to  be 
experienced  enough  to  navigate 
the  ship  with  RADAR  only. 

Much  the  same  as  in  the  runs 
with  traditional  navigation,  there 
were  significant  differences 
between  the  three  experience 
levels. 

Masters  performed  slightly  better 
than  the  pilots,  mates,  and  cadets 
while  navigating  with  ECDIS,  as 
seen  in  Figure  2.  They  seemed  to 
have  a  slight  edge  in  shiphandling 
of  the  particular  vessel,  the  TA^ 
KINGS  POINTER.  The  masters 
seemed  to  be  able  to  anticipate 
turns  better,  more  so  than  the 
pilots,  who  were  accustomed  to 
larger  ships,  and  the  mates  who 
do  not  often  perform 
shiphandling  in  confined 
waterways. 

■3.3.2  Track  Keeping 
Performance 

Preliminary  analysis  with  the 
ANOVA  method  indicated  that 
cross  track  distance  tended  to 
decrease  with  the  run  number. 
This  may  be  due  to  learning 
effect,  since  some  of  the 
mariners  performed  the  same  run 
more  than  once  under  different 
conditions.  A  more  thorough 
analysis  showed  that  the  main 
variable  which  explained 
variation  in  cross  track  distance 
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was  navigation  type.  In  general,  mariners  using  ECDIS  had  significantly  lower  cross  track 
distance  than  those  using  RADAR  or  traditional  techniques.  Also,  those  mariners  who  used 
traditional  techniques  performed  better  than  those  who  used  only  the  RADAR. 

3.3.3  Day  and  Night  Comparisons 

Another  variable  which  explains  cross  track  distance  variation,  in  conjunction  with  navigation 
type,  is  day  vs.  night.  As  was  expected,  the  mariners  tested  during  day  runs  performed  better 
than  those  tested  during  night  runs. 

The  interaction  between  navigation  type  and  day  vs.  night  is  interesting.  On  day  runs,  mariners 
using  ECDIS  performed  the  best,  with  little  performance  difference  between  those  using 
RADAR  only  and  those  using  traditional  methods.  At  night,  mariners  using  ECDIS  still 
performed  the  best,  and  those  using  traditional  and  RADAR  methods  performed  considerably 
worse. 

The  direction  of  transit  did  not  have  any  significant  effect  on  navigation  performance. 

3.3.4  Responses  from  Evaluations 

Evaluation  forms  were  filled  out  by  16  of  the  20  test  subjects  (four  cadets  did  not  respond).  The 
evaluation  contained  48  true/false,  multiple  choice  and  fill  in  questions. 

A  compilation  of  the  results  showed  that  the  mariners  felt  differently  about  the  number  of  chart 
windows  displayed  at  one  time:  45%  wanted  a  two-chart  windows,  36%  called  for  a  one-chart 
window,  and  1 8%  wanted  a  three-chart  window.  When  asked  about  which  chart  orientation  they 
would  use  the  most,  93%  said  North-up.  For  most  important  sensor  displays,  the  mariners  chose: 
gyro,  course  made  good,  speed,  and  cross  track  distance.  Some  of  the  comments  which  were 
reported  were  concerns  about  having  no  traffic  information  (i.e.  RADAR  targets)  integrated  into 
the  ECDIS,  while  others  said  ECDIS  made  the  task  of  navigation  more  relaxing. 

3.3.5  Resulting  Contribution  to  ECDIS  Performance  Standards 

The  USCG IMO/IHO  HGE  representative  for  ECDIS  standards  development  used  the  findings 
from  these  sea  trials  to  develop  a  position  paper  on  several  of  the  ECDIS  Provisional 
Performance  Standards  Issues.  This  position  paper  along  with  other  Member  Nations 
recommendations  were  combined  to  create  an  IHO/IMO  HGE  Draft  Performance  Standard  for 
ECDIS  (IHO/IMO,  Sep  1992). 

4.0  SEPTEMBER  -  OCTOBER  1992  SIMULATOR  EXPERIMENTS 

Objective 

The  objectives  of  the  experiment  were  to  examine  several  broad  issues  underlying  the  IMO 
Draft  Performance  Standard,  for  which  the  simulator  was  especially  appropriate  as  a  tool.  The 
simulator  makes  it  possible  to  examine  the  dynamic  situation  of  route  monitoring  with  a 
control  that  would  be  difficult  or  impossible  at  sea.  The  following  four  major  issues  were 
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addressed: 


.  rnntrihiition  of  F.CDTS  to  the  Safety  of  Navigation.  ECDIS  should  enhance  safety  by 
affording  the  mariner  a  more  accurate  knowledge  of  the  ship’s  position,  and  its  relation 
to  a  planned  route  and  to  potential  hazards  than  is  possible  with  conventional  bridge 
procedures  and  a  paper  chart. 

.  T^ediietinn  of  Navigational  Workload  bv  ECDIS.  ECDIS  can  integrate  information 

from  a  number  of  sensors,  and  can  automate  the  primary  and  generally  time-consuming 
navigation  function  of  position  fixing.  These  capabilities  should  reduce  the  mariner's 
workload.  There  is,  of  course,  the  implicit  assumption  that  reduced  workload  also 
contributes  to  greater  safety. 

•  Chart  Feamres  and  Navigational  Functions  on  ECDIS.  At  this  early  point  in  the 
development  of  ECDIS  technology,  there  was  no  industry  consensus  about  which 
electronic  chart  features  and  computer-based  navigation  functions  would  be  needed  by 
mariners.  The  simulator  experiment  allowed  observation  of  mariners’  use  and  selection 
of  features  and  functions.  This  use  was  observed  while  mariners  operated  two  systems 
under  a  variety  of  marine  conditions. 

.  Tntepration  of  RADAR  Feamres  on  ECDIS.  The  most  highly  integrated  navigational 
system  combines  two  plan  view  displays  —  the  electronic  navigation  chart  and 
RADAR/automated  RADAR  plotting  aid  (ARPA)  -  on  one  system.  It  was 
hypothesized  that  this  integration  would  have  positive  effects  on  safety  and  workload. 

4.1  EXPERIMENTAL  DESIGN 

Subjects 

The  six  participants  for  the  study  included  four  masters  (unlimited  license),  each  with  more  than 
20  years  of  experience,  one  chief  mate  and  one  second  mate.  During  each  week,  each  mariner 
received  brief  but  formal  training  on  the  two  ECDIS  systems  and  ran  through  all  the 
experimental  scenarios  in  a  different  counter-balanced  order. 

Method 

The  test  plan  was  intended  to  maximize  the  advantages  of  a  simulator  by  selecting  those  issues 
most  effectively  and/or  efficiently  examined  there.  As  is  frequently  done  in  simulator  rese^ch, 
the  workload  was  increased  beyond  realistic  levels  on  the  assumption  that  a  high,  but  sustainable, 
workload  increases  the  sensitivity  of  the  performance  measures.  To  ensure  a  high  workload, 
each  participating  mariner  made  port  arrivals  and  departures  as  the  one  officer  alone  on  the 
bridge.  In  addition,  no  pilot  came  on  board  when  the  ship  passed  the  pilot  station.  To  keep  the 
workload  sustainable,  the  equipment  consoles  were  arranged  for  “centralized  control”  to 
minimize  movement  around  the  bridge.  A  qualified  helmsman  was  present  in  all  scenarios. 

As  the  single  officer  on  the  bridge,  the  subject  mariner  was  responsible  for  navigation,  collision 
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avoidance,  and  bridge  management  activities.  The  experimental  seenarios  were  designed  to  be 
approximately  equal  in  level  of  difficulty  representing  eaeh  of  these  three  categories  of  activities. 
This  scenario  design  is  an  example  of  the  type  of  control  that  is  possible  on  the  simulator  but  not 
at  sea. 

The  simulation  tests  consisted  primarily  of  route  monitoring  in  the  relatively  high  risk  coastal 
and  harbor/harbor  approach  phases  of  navigation  during  vessel  arrivals  and  departures  from  port 
(Federal  Radionavigation  Plan,  1990).  Each  mariner  spent  a  full  week  on  the  simulator,  making 
harbor  transits  with  ECDIS  configurations  that  differed  in  such  critical  features  as  position 
updating  mode  and  RADAR  overlay. 

The  primary  eontrolled  factor  was  the  method  of  navigation  available  to  the  watchstander  in  a 
given  scenario  (Table  4.1.1).  The  conventional  choices  for  navigation  were:  position  fixing  on 
the  paper  chart,  RADAR/ARPA,  and  visual  piloting.  In  two  “baseline”  scenarios,  only  these 
conventional  methods  were  available.  In  the  remaining  scenarios,  one  of  the  commercial 
systems  was  added  to  the  bridge  in  one  of  three  modes: 

.  ECDIS  with  automatic  position  updating  and  RADAR  features  (Positioning  was 
to  an  accuracy  of  five  meters  or  better.  Mariners  were  told  that  differential  Global 
Positioning  System  was  in  use.) 

•  ECDIS  with  automatic  position  updating  and  ns  RADAR  features 

•  ECDIS  vnthout  automatic  position  updating  (and  with  instructions  to  update 
manually) 

In  addition,  the  experimental  scenarios  were  designed  to  vary  in  visibility,  route,  and  harbor. 


Table  4.1.1  Experimental  Design  Factors 


- 1 - 

Factors  Controlled 

Levels 

1)  Navigation  Type 

Traditional  Navigation:  visual  piloting,  paper  chart,  RADAR/ARPA 

ECDIS  with  automatic  position  updating  and  RADAR  features  (Positioning  was  to  an 
accuracy  of  five  meters  or  better.  Mariners  were  told  that  differential  Global 

Positioning  System  was  in  use.) 

ECDIS  with  automatic  position  updating  and  no  RADAR  features 

ECDIS  without  automatic  position  updating  (and  with  instructions  to  update  manually) 

2)  Visibility 

Unlimited 

Clear 

Reduced  to  one  nautical  mile 

3)  Route 

Inbound 

Outbound 

4)  Harbor 

New  York 

San  Francisco 
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A  variety  of  human  factors  techniques  for  measuring  performance  and  operator  reactions  were 
used  (Table  4.1.2).  These  included:  measures  of  workload  using  the  National  Aeronautics  and 
Space  Administration's  Task  Load  Index  -  NASA  TLX  (Hart  and  Stayeland,  1988),  situational 
awareness,  ratings  of  safety  and  preference,  and  extensive  questionnaires. 


Table  4.1.2  Experimental  Design  Performance  Measures 


Objective  Measures 

1)  Cross  Track  Distance 

2)  Situational  Awareness 

3)  Ratings  of  Safety  and  Preference 

Subjective  Measures 

1)  Extensive  Questionnaires 

2)  NASA  TLX 

4.2  INSTRUMENTATION 

The  experiment  was  run  at  MSI/C AORF  in  Kings  Point,  New  York.  This  simulator  has  a 
realistically  equipped  full  mission  bridge  and  a  considerable  history  of  human  factors  and  ship 
control  research.  The  simulator  capabilities  include  sophisticated  ship  models,  harbor  data  bases, 
observational  and  data  collection  methods. 

During  the  selection  of  ECDIS  devices,  the  LFSCG  chose  two  commercially  available  ECDIS 
units  which  most  closely  met  the  requirements  for  this  evaluation  (Table  4.2.1). 


Table  4.2.1  Instrumentation 


Experimental 

Platform 

Marine  Safety  Intemational/Computer  Aided  Operations  Research  Facility  (MSI/CAORF)  in 
Kings  Point,  New  York 

ECDIS  Platform 

1)  Precision  Integrated  Navigation  System  -  VME  (PINS-VME):  Offshore  Systems  Ltd 
roSI.V  Canada  -  Comment:  nrovided  proven  RADAR  overlay. 

2)  Disk  Navigation  System  (ROB),  Robertson  Marine  Systems  Inc. ,  Norway  - 
Comment:  electronic  chart  system  more  IMO  compliant. 

Common  Features: 

vector  based  digital  charts  data  produced  by  its  manufacturer,  which  allows  for  fast  chart 
updating  and  extensive  "zooming"  capabilities 

position  sensors  used  -  ship's  gyro  compass  and  Differential  Global  Positioning  System 
(DGPS) 

waypoint  navigation  capability 

displays  for  course  over  ground  (COG),  speed  over  ground  (SOG),  cross  track  distance  (XTD), 
latitude  and  longitude  (lat/lon),  and  range  and  bearing 

capability  to  set  alarms  (i.e.  entering  a  mariner  determined  danger  area) 

integrated  RADAR  overlay 

Comment 

user  interface  advanced,  allows  for  custom  screens  and  display  of  sensor  information; 
navigation  menu  very  user-friendly  and  intuitive 
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These  two  systems  differed  from  each  other  in  their  treatment  of  such  features  as  chart 
presentation,  RADAR  overlay  and  user  interface.  The  Offshore  Systems  Ltd.’s  (OSL), 

Precision  Integrated  Navigation  System  -  VME  (PINS-VME)  provided  a  proven  RADAR 
overlay  capability.  The  Robertson  Marine  Systems,  Inc.’s  Disk  Navigation  System  (ROB), 
most  closely  conformed  to  the  IMO  Provisional  Performance  Standards  (IMO,  1989)  for  display 
presentation  and  functions.  Both  systems  were  interfaced  with  the  MSI/CAORF  simulator. 

4.3  RESULTS 

4.3.1  Primary  Method  of  Navigation 

The  primary  method  of  navigation  that  each  mariner  used  for  each  identifiable  segment  of  the 
transit  was  reported  by  the  mariner  after  each  scenario.  The  assumption  was  that  the  mariner’s 
selection  of  method  would  reflect  his  view  of  the  best  combination  of  safety  and  workload  for  the 
conditions.  The  results  are  summarized  in  Table  4.3.1  for  the  harbor/harbor  approach  phase  of 
navigation,  with  its  high  risk  and  high  workload. 

The  test  subjects  rarely  reported  using  the  paper  chart  for  position  fixing.  Instead,  during 
conventional  bridge  conditions  (i.e.,  no  ECDIS  available),  visual  piloting  and  RADAR/ARPA 
were  reported  to  have  been  used.  When  ECDIS  with  automatic  updating  of  position  was 
available,  it  was  reported  to  be  the  predominant  method  of  navigation.  Without  automatic 
updating  of  position  and  with  the  requirement  to  update  manually,  ECDIS  lost  its  preferred 
status. 


Table  4.3.1  Proportion  of  Total  Segments  for  Which  Each  Method  of  Navigation 
_ Was  Reported  as  Primary  in  Harbor/Harbor  Approach  Navigation 


- - - -X. - 

BRIDGE  CONDITIONS 

PLOTTING/ 

PAPER 

RADAR/ 

ARPA 

VISUAL 

PILOTING 

ECDIS 

TOTAL  # 
SEGMENTS 

Conventional  Bridge 

0.03 

0.25 

0.73 

NA 

40 

ECDIS  Auto  Positioning 

0.00 

0.15 

0.18 

0.67 

79 

ECDIS  no  Auto  Positioning 

0.05 

0.61 

0.28 

0.05 

18 

4.3.2  Safety  Measured  bv  Accuracy  of  Track  Keeping 

Safety  of  navigation  has  been  measured  in  simulator  research  (Kaufinan,  1985;  Schryver,  1985) 
and  in  sea  trials  (Gonin  and  Crowell,  1992)  by  cross  track  distance  from  a  planned  route. 
Although  no  special  instructions  to  keep  the  ship  close  to  the  line  were  given  in  this  experiment, 
it  was  hypothesized  that  ECDIS  would  increase  safety  by  reducing  cross  track  distance.  At 
some  critical  points,  such  as  approaches  to  bridges  or  to  major  turns,  the  availability  of  ECDIS 
with  automatic  positioning  resulted  in  substantial  reductions  in  mean  cross  track  distances. 
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4.^.^  WorklnaH  Measured  hv  Time  Spent  and  bv  Mariner’s  Ratings 


Workload  was  measured  after  each  scenario  by  asking  the  mariner  what  proportion  of  his  time 
was  spent  on  navigation,  collision  avoidance  and  bridge  management.  A  self-reported  workload 
on  each  of  these  three  tasks  was  also  administered  using  the  NASA-TLX  (Hart  and  Staveland, 
1988). 

A  summary  of  the  findings  are  presented  in  Table  4.3.2.  The  availability  of  ECDIS  with  auto¬ 
positioning  decreased  both  the  mean  workload  for  navigation  and  the  mean  reported  proportion 
of  time  spent  on  navigation,  as  compared  to  that  measured  for  conventional  bridge  procedures. 
The  necessity  of  manually  updating  position  increased  the  navigation  workload  and  the 
proportion  of  time  spent  on  navigation.  Workload  was  increased  over  that  for  both  the 
conventional  bridge  and  ECDIS  with  automatic  positioning. 


Table  4.3.2  Mean  Navigation  Workload  and  Reported  Distribution  of  Mariner's  Time 


BRIDGE 

CONDITIONS 

NAVIGATION 

WORKLOAD 

NAVIGATION 

COLLISION 

AVOIDANCE 

BRIDGE 

MANAGEMENT 

Conventional  Bridge 

52 

0.46 

0.33 

0.21 

ECDIS  Auto  Positioning 

36 

0.37 

0.41 

0.21 

ECDIS  NO  Auto- 
Positioning 

63 

0.49 

0.34 

0.17 

4.3.4  Time  Spent  on  Navigation  Versus  Time  Spent  on  Collision  Avoidance 

Table  4.3.2  also  shows  that  with  the  decrease  in  proportion  of  time  spent  on  navigation  using 
ECDIS  with  automatic  positioning,  there  was  a  corresponding  increase  in  the  proportion  of  time 
spent  on  “look  out”  and  on  collision  avoidance.  In  the  mariner's  view,  this  shift  represented  an 
increase  in  safety. 

Navigation  workload  and  the  proportion  of  time  spent  on  navigation  were  positively  and 
significantly  correlated  with  each  other.  Navigation  workload  and  the  proportion  of  time  spent 
on  collision  avoidance  were  negatively  and  significantly  correlated. 

4.3.5  F eature  and  F unction  U se 

The  use  of  chart  features  and  navigation  functions  on  ECDIS  was  examined  in  a  number  of  ways. 

•  experimental  observers  watched  on  video  monitors  and  tallied  features  and  functions 
enabled  on  the  ECDIS  systems  by  the  mariner 

•  questionnaires  after  each  scenario  contained  checklists  for  reports  of  what  had  been  used 
and  what  was  wanted 

•  a  final  questionnaire  contained  a  checklist  asking  the  mariner  to  recommend  what  should 
be  available 
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The  results  obtained  with  this  final  measure  showed  that  only  a  relatively  small  set  of  features 
were  recommended  by  a  majority  of  the  mariners  as  “display  always.”  The  mariners  instead 
showed  a  clear  preference  that  all  objects  commonly  found  on  a  navigation  chart  should  be 
available  at  user’s  option.  Mariners  indicated  that  although  a  simple  display  is  preferred  for  the 
monitoring  task,  other  more  detailed  information  should  be  readily  available. 


TabIe^3j3^Dis£laj^Jnfbrmation^lostJ|refer^^ 


FEATURES 

DISPLAY  ALWAYS 

AT  USERS’S  OPTION 

Charted  Features 

Coastline/Landmass 

100% 

0% 

Fixed  Aids  to  Navigation 

100% 

0% 

Floating  Aids  to  Navigation 

100% 

0% 

Federal  Channel  Lines 

67% 

33% 

Navigation  Lanes/Fairways 

67% 

33% 

Pilot  Areas 

67% 

33% 

Indication  of  Isolated  Dangers 

67% 

33% 

ECDIS  Generated  Features 

Ownship  Outline 

83% 

17% 

Display  Planned  Trackline 

83% 

17% 

Navigation  Fault  Alarms  (i.e., 
GPS  down) 

100% 

0% 

4.3.6  Results  on  the  Use  of  RADAR  Overlay 

No  significant  differences  were  found  between  ECDIS  with  and  without  RADAR  features,  or 
between  ECDIS  with  the  complete  RADAR  video  and  ECDIS  with  targets  only.  Mariners  felt 
that  RADAR  integration  should  be  a  valuable  addition  to  ECDIS.  The  systems  they  used, 
however,  were  not  good  examples  of  how  RADAR  integration  should  work.  The  principal 
drawbacks  mentioned  were  an  overly  cluttered  screen  and  incomplete  ARPA  information  that  did 
not  allow  them  to  depend  on  the  ECDIS  for  both  navigation  and  collision  avoidance. 
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‘^.n  JAN-  APR  199^  RTTTERSWEET& KINGS  POINTER  SEA  TRIALS 


Objective 

Since  these  tests  were  meant  to  be  an  at-sea  verification  of  the  simulator  results,  the  objectives  of 
the  research  remained  very  similar. 

•  Contribution  of  ECDIS  to  the  Safety  of  Navigation 

.  Reduction  of  Navigation  Workload  by  ECDIS 

•  Chart  Features  and  Navigation  Functions  on  ECDIS 

•  Integration  of  RADAR  Features  on  ECDIS. 

5.1  EXPERIMENTAL  DESIGN 
Subjects 

There  were  two  pools  of  subjects  for  these  tests.  For  the  first  part  of  the  experiment,  all  of  the 
BITTERSWEET'S  seven  qualified  Officers  of  the  Deck  (OOD's)  participated,  three  senior 
watchstanders  and  four  junior  watchstanders.  In  the  second  part,  thirteen  experienced  mariners 
participated:  four  pilots,  four  masters  and  five  mates. 

Method 

The  experiment  consisted  of  two  parts.  The  first  part  was  performed  aboard  the  USCG  Cutter 
BITTERSWEET.  It  allowed  for  a  direct  comparison  of  navigational  performance  on  an  ECDIS 
equipped  bridge  with  traditional  navigation  using  the  paper  chart.  The  conceptual  approach  to 
this  at-sea  test  was  to  examine  baseline  performance  with  the  mariner  using  traditional 
techniques:  a  paper  chart,  RADAR/ARPA,  and  visual  piloting.  Then,  ECDIS,  in  two  different 
modes,  was  used  to  examine  its  effect  (i.e.,  the  changes  from  baseline  performance).  In  addition, 
such  v^iables  as  area  of  transit,  visibility,  time  of  day  and  mariner  were  evaluated. 

The  second  part  of  the  experiment  was  performed  aboard  the  IN  KINGS  POINTER.  This  part 
of  the  experiment  was  designed  to  gather  additional  data  and  information  in  specific  areas  which 
did  not  provide  enough  conclusive  evidence  in  the  first  part  of  the  experiment  aboard  the 
USCGC  BITTERSWEET. 

A  variety  of  human  factors  techniques  for  measuring  performance  and  operator  reactions  were 
used.  These  included  measures  of  workload  such  as  the  U.S.  National  Aeronautics  and  Space 
Administration's  Task  Load  Index  (NASA  TLX,  1988)  ratings  of  safety  and  preference, 
comprehensive  user  questionnaires  and  interviews  (Table  5.1.1). 

Tests  scenarios  were  performed  in  Buzzards  Bay  and  Narragansett  Bay.  These  test  included 
operations  in  coastal  and  confined  waterways  during  periods  of  good  and  poor  visibility,  and 
during  day  and  night  transits  with  low  and  moderate  traffic  density  using  various  navigation 
methods  (Table  5.1.2). 
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Table  5.1.1  Experimental  Design  Performance  Measures 


Objective  Measures 

1)  Cross  Track  Distance 

2)  Course  made  good  (CMG) 

3)  Gyrocompass  heading 

4)  Speed  Over  Ground  (SOG), 

5)  Speed  Made  Good  (SMG), 

Subjective  Measures 

1)  Extensive  Questionnaires 

2)  NASA  TLX 

Factors  Controlled 

Levels 

1)  Navigation  Type 

ECDIS  with  RADAR  overlay  -  this  mode  was  chosen  to  allow  for  evaluation  of  the 
IMO  Draft  Performance  Standard  for  the  integration  of  RADAR  into  ECDIS. 

ECDIS  without  RADAR  overlay  -  this  mode  was  chosen  since  it  would  most  likely 
be  the  base  configuration  for  ECDIS  and  would  allow  for  evaluation  of  many 
mariner-ECDIS  interface  issues. 

Traditional  navigation  with  paper  chart  -  this  mode  will  be  the  baseline  from  which 
to  compare  navigational  performance  with  performance  using  ECDIS. 

ECDIS  without  automatic  nosition  updating  and  with  instructions  to  update  manually 

2)  Visibility 

Good  -  greater  than  one  mile  visibility 

Poor  -  less  than  one  mile  visibility  or  simulated  zero  visibility  (i.e.  eovered  bridge 
windows). 

3)  Time  of  Day 

Day 

Night 

4)  Area  of  Operation 

Coastal 

Harbor/Harbor  Approach 
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Table  5.1.3-  Sea  Trial  Scenario  Matrix 


Test 

No. 

Test  Area 

Navigation 

Method 

Condition 

Visibility 

Condition 

Time 

Number  of  Runs 
Performed 

1 

A 

ECDIS  w/RADAR 

Good 

Day 

2 

2 

COASTAL 

Night 

0 

3 

Poor 

Day 

2 

4 

Night 

2 

5 

ECDIS  wo/RADAR 

Good 

Day 

3 

6 

Night 

2 

7 

Poor 

Day 

2 

8 

Night 

1 

9 

TRADITIONAL 

Good 

Day 

I 

10 

Night 

1 

11 

Poor 

Day 

2 

12 

Night 

3 

13 

B 

ECDIS  w/RADAR 

Good 

Day 

2 

14 

HARBOR/ 

Night 

0 

15 

HARBOR 

Poor 

Day 

2 

16 

APPROACH 

Night 

2 

17 

ECDIS  wo/RADAR 

Good 

Day 

2 

18 

Night 

2 

19 

Poor 

Day 

3 

20 

Night 

2 

21 

TRADITIONAL 

Good 

Day 

3 

22 

Night 

1 

23 

Poor 

Day 

1 

24 

Night 

2 

N' 

OTE:  Test  scenario  matrix  was  intended  to  be  performed  twice.  TOTAL  =  43 

5.2  INSTRUMENTATION 

5.2. 1  Test  Vessel  -  USCGC  BITTERSWEET 

Sea  trials  were  conducted  during  January  -  February  1993  aboard  the  USCG  Cutter 
BITTERSWEET  (WLB-389).  This  180'  (55m)  ship  was  built  in  1944,  with  a  37'  (1 1.3m)  beam, 
13'  (4m)  draft,  and  1025  dead  weight  ton  (dwt)  displacement.  This  vessel  was  particularly  suited 
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for  test  and  evaluation  of  ECDIS  because  of  the  crews'  familiarity  with  advanced  shipboard 
navigation  systems  such  as  GPS,  differential  GPS(DGPS),  Laptop  Automated  Aids  Positioning 
System  (LAAPS),  and  Electronic  Chart  Systems  (ECS)  (Alexander  and  Quinn  1992). 

Normal  ship's  complement  onboard  USCGC  BITTERSWEET  is  45  enlisted  personnel  and  seven 
officers.  During  the  trials,  there  were  seven  qualified  Officers  of  the  Deck  (OOD).  Six  were 
commissioned  officers,  and  one  first  class  petty  officer  quartermaster.  In  terms  of  experience, 
there  were  three  senior  watchstanders  and  four  junior  watchstanders. 


5  2.2  Te.st  Ves.se1  -  T/V  KINGS  POINTER 

A  new  T/V  KINGS  POINTER  was  received  by  the  USMMA  from  the  Navy  in  1992.  This 
training  ship  is  the  former  U.  S.  Naval  Ship  CONTENDER  (T-AGOS  2)  built  in  1984.  This 
vessel  has  a  overall  length  of  224'  (67m),  beam  of  43'  (13m),  draft  of  15'  (4.5m)  and  dead  weight 
displacement  of  2,250  tons.  The  location  of  this  vessel  in  the  mouth  of  New  York's  East  River  in 
Long  Island  Sound  makes  an  ideal  area  for  coastal  and  harbor  transits.  A  total  of  1 1  mariners  (5 
mates,  4  masters  and  2  pilots)  from  the  USMMA  and  nearby  State  University  of  New  York 
Maritime  College  participated  in  the  experiment  as  well  as  two  pilots  from  the  Northeast  Pilots 
Association. 


5.2.3  Test  Equipment 

The  Offshore  Systems  Limited  (OSL)  Electronic  Chart  Precise  Integrated  Navigation  System 
(ECPINS)  was  the  primary  ECDIS  used  for  the  sea  trial  evaluation  of  the  IMO/IHO  ECDIS  Draft 
Performance  Standard.  With  the  exception  of  implementing  the  new  colors  and  symbols  recently 
developed  by  an  IHO  Colors  and  Symbols  Working  Group,  this  Canadian-manufactured  ECDIS 
system  was  nearly-compliant  with  the  current  IMO  Draft  Performance  Standard  published  by 
HGE  13  in  September  1992  (IMO/IHO  HGE  1992). 

In  order  to  give  the  watchstanders  an  opportunity  to  learn  about  ECDIS,  the  PINS  9000 
manufactured  by  Offshore  Systems  Limited  (Canada)  was  installed  aboard  USCGC 
BITTERSWEET  in  January  of  1992  and  used  by  the  mariners  for  1 1  months.  In  December 
1992,  a  new  Offshore  Systems  Limited  ECDIS  called  the  Electronic  Chart  Precise  Integrated 
Navigation  System  (ECPINS)  was  installed  specifieally  for  this  evaluation.  Two  months  later, 
the  same  version  of  ECPINS  was  installed  aboard  the  T/V  KINGS  POINTER.  Also,  at  that  time, 
two  other  ECDIS  devices  were  installed;  the  van  Rietschoten  en  Houwens  (R&H)  Hydrant 
Electronic  Chart  (EC)  Series  and  Atlas  Electronik  ATLAS  9800  ECDIS.  (Table  5.2.1) 
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Table  5.2.1  Instrumentation 


Experimental  Platform 

A)  USCG  Cutter  BITTERSWEET  (WLB-389) 

B)  T/V  KINGS  POINTER  (formerly  CONTENDER:  T-AGOS  2) 

ECDIS  Platform 

1)  Electronic  Chart  Precise  Integrated  Navigation  System  (ECPINS)  :  Offshore 

Systems  Ltd  (OSL),  Canada  (Installed  on  Platform  A  and  B) 

2)  van  Rietschoten  en  Houwens  (R&H)  Hydraut  Electronic  Chart  (EC)  Series  (B  Only) 

3)  Atlas  Electronik  ATLAS  9800  ECDIS  (B  Only). 

Features: 

1,  2  &  3)  vector  based  digital  charts;  chart  data  produced  by  its  manufacturer,  which 
allows  for  fast  chart  refresh,  and  extensive  "zooming"  capabilities. 

1,  2  &  3)  used  the  ship's  gyro  compass  and  Differential  Global  Positioning  System 
(DGPS) 

1  &  2)  capable  of  waypoint  navigation 

1,  2  &  3)  displays  for  course  over  ground  (COG),  speed  over  ground  (SOG),  cross 
track  distance  (XTD),  latitude  and  longitude  (lat/lon),  and  range  and  bearing 

1  &  2)  capability  of  setting  alarms  (i.e.  entering  a  mariner-determined  danger  area) 

2)  capability  to  input  range  and  bearing  from  objects  for  position  checking 

Comments 

2)  IHO  standard  chart  data,  excellent  detail  and  representation  of  chart  information 

1  &  2)  user  interface  advanced,  allows  for  custom  screens,  and  integration  of  many 
sensor  displays;  menu  strucmre  user-friendly,  and  somewhat  intuitive 

1)  good  integrated  RADAR  Overlay 

3)  excellent  RADAR  picmre  with  full  RADAR/ARPA  functionality 

5.3  RESULTS 

5.3.1  Performance  Measures  and  Data  Analysis 

Performance  measures  and  data  analysis  procedures  were  developed  to  determine  the  effect  of 
ECDIS  on  navigation  safety,  and  what  ECDIS  features  and  functions  were  essential  to  insuring 
navigation  safety  and  efficiency.  Navigation  safety  was  directly  addressed  by  the  use  of 
measures  of  cross  track  distance  and  mariner  performance.  Navigation  safety  and  the 
efficiency  of  ECDIS  were  indirectly  addressed  by  measuring  the  mariner's  workload.  The 
question  of  what  ECDIS  features  and  functions  were  essential  to  safe  and  efficient  navigation 
was  addressed  by: 

•  recording  the  mariners  use  of  ECDIS 

•  recording  the  features/functions  the  mariners  sought  (when  not  available  on  the 
ECDIS) 
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•  obtaining  the  mariner's  expert  opinions  regarding  both  general  and  specific 
aspects  of  the  design  and  use  of  ECDIS 

5.3.2  Effects  on  Safety  and  Workload 

The  experiment  was  designed  to  test  the  hypotheses  that  ECDIS  would  reduce  the  mariner’s 
workload  for  navigation.  With  more  available  time  and  attention  for  other  mariner  functions, 
an  indirect  benefit  should  be  increased  safety.  Using  the  NASA  TLX,  mariners  were  asked  to 
rate  workload  after  each  run  and  to  report  the  proportion  of  time  spent  performing  navigation 
tasks,  collision  avoidance  tasks,  and  bridge  management  tasks. 

As  shown  in  Table  5.3.1,  when  using  ECDIS  with  RADAR  overlay,  the  average  workload 
reported  by  the  mariner  was  the  lowest.  Additionally,  the  mariner  spent  a  smaller  proportion 
of  his  time  on  the  navigation  task,  leaving  more  time  to  spend  on  the  higher  risk  collision 
avoidance  task.  Compared  to  more  traditional  means  of  navigation,  the  use  of  ECDIS  can 
significantly  reduce  navigation  workload. 


Table  5.3.1  -  Navigation  Workload  and  Di^ribution  of  Mariner's  Time 


Bridge  Conditions 

Navigation 
Work  Load® 

Proportion  of  Time*^  || 

Navigation 

Collision 

Avoidance 

Bridge 

Management 

Traditional  Bridge^ 

29 

65 

23 

12 

ECDIS 

22 

63 

24 

13 

ECDIS  w/RADAR  Overlay' 

17 

61 

26 

13 

Notes 

a.  Mean  mariner  rating  of  navigation  w< 

b.  Mean  mariner  report  of  proportion  o1 

c.  Workload  measure  during  navigation 

3rkload  using  NASA-TLX 
f  time  spent  on:  navigation 
with  ECDIS  using  RADA 

,  collision  avoidance, 
R  Overlay  is  statistica 

and  bridge  management, 
lly  different  from  traditional  bridge  at  0.05  level 

5.3.3  Display  Features 

The  IMO/IHO  Draft  Performance  Standard  for  ECDIS  1992  (Draft  Performance  Standard) 
defines  three  subcategories  of  information  display  for  the  ECDIS. 

o  Base  Display 

"The  permanent  level  of  display  information  that  would  be  required  at  all  times,  in  all 
geographic  areas,  and  under  every  circumstance.  Not  intended  to  be  sufficient  for  safe 
navigation;  the  mariner  is  expected  to  add  System  Electronic  Navigational  Chart  (SENC) 
information  as  required  for  the  situation  at  hand. " 

o  Standard  Display 

"The  level  of  display  information  available  when  a  chart  is  first  displayed,  which, 
depending  on  the  needs  of  the  mmner,  can  be  modified;  but  which  is  always  accessible 
again  by  a  single  operator  action." 
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o  All  Other  Information 

"Any  information  which  the  mariner  uses  to  navigate  which  isn't  covered  by  the  previous 
two  categories." 


The  participating  mariners  were  given  a  list  of  seventy-nine  chart  features,  including  the  chart 
features  that  the  Draft  Performance  Standard  requires  for  base  and  standard  display.  The 
tabulated  responses  are  discussed  in  the  next  section. 

Base  Display 

For  the  base  display,  the  Draft  Performance  Standard  list  of  features  is  more  comprehensive  than 
that  chosen  by  the  mariners.  Figures  7  and  8  show  three  lists  of  features.  The  top  right  area  of 
the  figures,  lists  the  features  the  Performance  Standard  recommends,  but  were  not  selected  by  the 
mariner.  The  top  left  area,  lists  the  features  the  mariners  wanted,  but  were  not  included  in  the 
Performance  Standard.  The  bottom  area  of  the  figures,  lists  those  features  common  to  both  the 
mariner  and  the  Performance  Standard.  Figure  7  illustrates,  four  out  of  the  five  features  (80%) 
are  common  features  chosen  by  the  mariner  and  are  included  in  the  Performance  Standard.  Three 
features  are  included  in  the  base  display  of  the  Performance  Standard  but  were  not  selected  by 
the  mariner.  Fixed  and  floating  aids  to  navigation,  the  only  feature  chosen  by  the  mariners  not 
on  the  Draft  Performance  Standard  base  display  list,  is  a  feature  of  the  Draft  Performance 
Standard's  standard  display. 


Features  Exclusive  to  Performance  Standard 

;  .  ■  ■  )  ~  units  of  depth/height 

~  —  scale/range/orientation 

■  :  —  traffic  routing  systems 


Features  Common  to  Mariner  &  Standard 

—  isolated  dangers 
~  indication  of  underwater  danger 

—  ownship  safety  depth  contour 

—  coastline 


Figure  7  -  Base  Display  Features  Summarized 


Feature  Exclusive  to  Mariner  Preference 


—  fixed/floating  navigation  aids 


Standard  Display 

For  the  standard  display,  the  mariners  proposed  a  larger  list  of  features  than  the  Draft 
Performance  Standard.  In  the  base  display,  the  smaller  list  was  for  the  most  part  (86%),  a  subset 
of  the  larger.  The  Draft  Performance  Standard's  standard  display  included  two  features  not 
found  in  the  mariner  preference  list:  visual  and  RADAR  conspicuous  features,  and  indication  of 
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cautionary  notices  (See  Figures  8).  It  is  interesting  to  note  that,  of  the  features  found  exclusively 
on  the  mariner  preference  list,  half  were  full  text  cautionary  and/or  information  notices.  This 
seemed  to  indicate  that  perhaps  mariners  had  selected  the  cautionary  notes  which  they  found 
most  useful  to  be  displayed  in  full,  instead  of  the  more  general,  less  informative  indications  of 
such  notices  required  by  the  Draft  Performance  Standard. 


Additionally,  four  ECDIS  functions:  ownship  outline,  heading/beam  bearing,  cursor  mark,  and 
display  planned  track  were  selected  by  the  mariner  to  be  available  in  the  standard  display  (i.  e. 
these  are  not  available  on  a  paper  chart).  This  tends  to  bear  out  the  findings  from  the  rest  of  the 
study,  which  indicate  that  the  mariners  felt  such  ECDIS  features  were  an  aid  to  navigation. 


Features  Exclusive  to  Performance  Standard 


Features  Exclusive  to  Mariner  Preference 


—  visual/radar  conspic  features 
---  ind  of  cautionary  notice 


anchor  area  info  notice 

—  channel  info  notice 

—  TSS  xing/rabout  notice 

—  TSZ  cautionary  notice 
---  cable  and  pipelines 


Features  Common  to  Mariner  &  Standard 

--  chart  scale  boundaries 

—  prohibited/restricted  area 

—  fain«ay/channels/etc. 

--  ind  fixd/float  nav  aids 

—  drying  line 
--  units  of  depth/height 

—  scale/range/orientation 

—  traffic  routing  systems 

—  ind  of  isolated  dangers 

—  ind  of  underwater  danger 

—  ownship  safety  depth  contour 

—  coastline 


Figure  8  -  Standard  Display  Features  Summarized 


5.3.4  Exit  Interview  on  Display  Features 

The  mariners  were  also  given  an  exit  survey  which  asked  questions  dealing  with  ECDIS 
functionality  beyond  the  chart  features  displayed.  Some  interesting  results  are  reported  here. 

An  overwhelming  majority  (92%)  of  the  mariners  felt  that  ECDIS  would  be  useful  as  a 
simulation/training  tool.  Although,  they  were  split  (58%  vs  42%)  on  whether  ECDIS  could 
function  as  primary  navigation  tool.  Most  of  the  mariners  the  training  they  received  was  not 
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enough.  They  also  reported  that  special  training  on  the  ECDIS  would  be  necessary,  similar  to 
RADAR  training. 

Most  mariners  felt  that  North-up  was  the  best  default  chart  orientation  for  route  planning,  open 
ocean  passage,  coastal  transit,  harbor  approach,  and  heavy  traffic.  However,  the  ability  to  switch 
from  one  orientation  to  another  seemed  to  be  a  popular  feature.  It  is  also  not  surprising  that  a 
majority  of  the  mariners  reported  that  ECDIS  symbols  should  always  look  the  same  as  paper 
chart  symbols. 

In  the  discussion  of  warnings,  the  mariners  prefer  both  audible  and  visual  alarms  to  be  given  in 
most  cases  of  likely  danger  (i.e.,  detection  of  possible  collision/grounding,  depth  less  than  safety 
depth  contour),  or  the  loss  of  position,  speed  or  other  essential  sensor  information.  For  warnings 
of  cross  track  error  and  entering  a  fishing  ground  or  restricted  area,  the  mariners  reported  that 
visual  alarms  were  sufficient. 

5.3  ..5  Statistical  Analysis  of  Test  Factors 

In  the  initial  set  of  statistical  analysis,  using  the  complete  data  set,  only  navigational  method 
had  a  significant  impact  in  improving  navigational  performance.  Next  a  second  set  of 
statistical  analysis  was  performed  using  only  the  data  from  the  ECDIS  runs,  differentiating 
between  those  with  and  without  RADAR  overlay.  In  these  second  tests  both  visibility  and  the 
presence  of  a  RADAR-overlay  were  significant  in  improving  navigational  performance. 

The  test  was  conducted  during  typical  New  England  January  weather.  From  these  tests,  it 
seems  quite  clear  that  ECDIS  and  ECDIS  with  RADAR-overlay  have  a  positive  impact  on 
navigational  performance  across  all  types  of  visibility,  times  of  day,  and  operators.  This  is 
encouraging,  not  just  for  the  proponents  of  ECDIS,  but  for  all  mariners. 

As  one  of  the  few  studies  providing  empirical  at-sea  evidence  to  determine  the  impact  of 
ECDIS  on  navigational  performance,  we  are  encouraged  that  our  findings  seem  to  corroborate 
the  findings  of  similar  studies  at  sea  (Gonin  and  Crowell,  1992)  and  at  ship  simulator  facilities 
(Smith,  et.  al,  1995,  and  Grabowski,  et.  al,  forthcoming). 

6.0  GENERAL  CONCLUSIONS 

The  tests  and  experiments  discussed  in  this  report  explored  a  progression  of  questions  about 
Electronic  Chart  Display  and  Information  Systems.  We  began  with  the  simple  questions  explored 
in  the  first  two  sea  trials :  Does  ECDIS  have  an  impact  on  overall  track  keeping  performance  ? 
What  are  the  effects  of  ECDIS  on  the  navigational  performance  of  mariners  with  different 
experience  levels?  These  questions  lead  to  more  complex  experimental  designs  of  the  simulator 
experiments  in  1992  and  the  1993  sea  trials. 

Results  of  1990  sea  trials  and  the  1991  KINGS  POINTER  trials  show  that  ECDIS  did 
significantly  improve  overall  track  keeping  performance,  and  that  there  were  no  significant 
differences  in  navigational  performance  between  mariners  with  varying  levels  of  experience 
when  navigating  with  ECDIS.  With  these  results,  we  can  certainly  answer  the  question:  Is 
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ECDIS  an  aid  to  navigation?  ",  in  the  affirmative.  The  ECDIS,  by  creating  a  real-time 
visualization  of  the  ship's  position  in  relation  to  its  surroundings,  aids  the  mariner.  This  provided 
important  groundwork  for  the  follow-on  experiments. 

The  objectives  of  the  1992  simulator  experiments,  as  well  as  the  1993  sea  trials  were  to  examine 
several  broad  issues  imderlying  the  IMO  Draft  Performance  Standards.  The  major  findings  of 
these  trials  are  discussed  here,  segmented  by  issue. 

Contribution  of  ECDIS  to  the  Safety  ofNavisation 

ECDIS  with  automatic  positioning  decreased  the  mean  cross-track  distance  to  approximately 
one  third  of  what  it  was  with  conventional  methods.  The  use  of  ECDIS  during  route  monitoring 
can  provide  equivalent  or  greater  safety  than  that  provided  by  the  use  of  the  paper  chart  and  more 
traditional  methods  of  navigation.  Three  mechanisms  which  provide  this  safety  were  identified: 

•  cross-track  distance  from  the  planned  route  is  decreased, 

•  mariners  of  all  skill  levels  can  perform  track  keeping  more  accurately  and 
efficiently, 

•  an  increased  proportion  of  the  mariners  time  is  spent  on  "look  out"  and  collision 
avoidance. 

These  findings,  that  ECDIS  supports  more  accurate  ship  control  and  allows  more  time  to  be 
spent  on  non-navigation  tasks,  agrees  with  simulator  evaluations  of  the  use  of  automation  for 
"one  man  bridge"  operations  (Schuffel,  Boer,  and  van  Breda,  1989).  In  a  time  where  increased 
tonnage  on  the  water  and  the  related  safety  issues  are  a  heated  topic  of  discussion,  these  results 
show  ECDIS  can  give  the  mariner  more  time  for  such  tasks  as  lookout  and  collision  avoidance, 
when  they  are  most  important.  It  must  be  made  clear  that  these  results  are  only  true  in  the  case 
of  ECDIS  with  automatic  positioning.  It  was  shown  in  these  sea  trials  that  in  the  case  of  a 
failure  of  the  automatic  positioning,  necessitating  manual  positioning  of  ECDIS,  cross-track 
distance  is  increased  and  ECDIS  loses  its  advantage  in  track  keeping  accuracy. 

Reduction  of  Navigational  Workload  bv  ECDIS 

In  addition  to  measures  of  cross-track  distance,  the  mariners  were  asked  to  report  on  their 
perceived  workload  with  and  without  the  ECDIS  present,  and  in  some  cases  with  and  without 
RADAR  overlay.  The  results  show,  as  was  suspected  from  the  reduction  in  cross  track  distance 
discussed  above,  mariners  reported  that  ECDIS  with  automatic  positioning  decreased  both  the 
mean  workload  for  navigation  and  the  mean  reported  proportion  of  time  spent  on  navigation.  It 
was  also  shovm  that  with  the  decrease  in  proportion  of  time  spent  on  navigation  using  ECDIS 
with  automatic  positioning,  there  was  a  corresponding  increase  in  the  proportion  of  time  spent  on 
look  out  and  collision  avoidance.  In  the  mariners'  view,  this  shift  represented  an  increase  in 
safety. 


Only,  in  the  later  at-sea  trial  when  using  ECDIS  with  advanced  RADAR 
integration  technology,  did  the  average  workload  reported  by  the  mariner  become 
the  lowest,  as  compared  to  when  the  mariners  had  an  ECDIS  with  more  primitive 
(i.e.  older)  RADAR  overlay. 
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•  Overall,  when  compared  to  more  traditional  means  of  navigation,  the  use  of 

ECDIS  with  automatic  positioning  can  significantly  reduce  navigation  workload. 
This  in  turn  will  give  the  mariner  more  time  to  devote  to  the  more  difficult  tasks 
of  look  out  and  collision  avoidance. 

.  Once  again  the  necessity  of  manually  updating  position  increased  the  navigation 
workload  and  the  proportion  of  time  spent  in  navigation. 

Chart  Features  and  Navigational  Functions  on  ECDIS 

Only  a  relatively  small  set  of  features,  (which  corresponded  very  closely  with  the  "base 
display”  in  the  Draft  Performance  Standards  for  ECDIS)  were  recommended  by  a  majority  of 
the  mariners  to  be  displayed  always.  The  larger  set  recommended  to  be  available  at  users 
option,  corresponded  very  closely  with  the  "standard  display”  in  the  Draft  Performance 
Standards  for  ECDIS. 

The  mariners  did,  however,  point  out  some  areas  where  the  Draft  Performance  Standard 
seemed  to  be  lacking: 

.  the  base  display,  or  at  the  very  least  the  standard  display,  should  require  a 
symbol  for  ownship's  position. 

.  the  Draft  Performance  Standard  needs  to  be  more  specific  about  when  and  how 
warnings  of  system  failures  should  be  presented. 

It  should  be  noted,  that  along  with  the  base  and  standard  displays,  the  participating  mariners 
indicated  a  need  for  an  immediate  and  easy  reference  to  a  much  larger  set  of  features  and 
functions  in  the  more  static  mode  of  route  planning.  This  capability  would  be  similar  to  a 
conventional  geographical  information  system  (GIS),  which  provides  much  more  extensive 
information  for  reference. 

Integration  of  RADAR  Features  on  ECDIS 

Although  mariners  in  the  1992  Simulator  Experiments  reported  that  RADAR  integration 
should  be  a  valuable  addition  to  ECDIS,  there  were  no  significant  differences  found  in  ECDIS 
with  and  without  RADAR  features.  This  is  most  likely  because  the  mariners  reported  that  the 
examples  that  they  saw  were  not  satisfactory.  The  principal  draw-backs  mentioned  were  an 
overly  cluttered  screen  and  incomplete  ARPA  information.  This  became  an  important  issue  in 
the  1993  sea  trials,  with  the  advances  in  technology  allowing  a  better  exploration  of  this  issue. 

In  the  1993  at-sea  tests,  as  the  advances  in  technology  allowed  a  truer  representation  of 
RADAR  integration,  there  was  a  decrease  in  workload  when  using  ECDIS  with  RADAR 
overlay  as  compared  to  traditional  methods  of  navigation.  The  comments  and  concerns  on 
collision  avoidance  expressed  by  most  of  the  mariners  in  these  studies  emphasize  the  need  to 
integrate  ECDIS  with  RADAR.  Mariners  indicated  a  preference  for  an  integrated 
ECDIS/RADAR/  ARPA  device  with  full  capabilities  for  navigation  and  collision  avoidance. 
This  integrated  system  would  have  to  be  completely  flexible  in  terms  of  which  information  is 
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available  at  a  particular  time. 

6.1  FUTURE  CONTRIBUTIONS  OF  ECDIS 

The  implications  and  impact  of  ECDIS  on  the  maritime  community  are  enormous.  Expected 
benefits  of  ECDIS  include  improvements  in  navigation  safety  and  increased  shipping 
efficiency;  all  of  which  mean  dollar  savings  to  national  marine  administrations  and  shipping 
companies.  Standards  which  describe  how  ECDIS  must  perform,  what  type  of  chart  data  it 
must  use,  and  how  this  data  are  to  be  displayed,  have  just  been  finalized.  After  many  years  of 
international  effort,  both  government  and  private,  these  standards  are  being  adopted  and  will 
be  held  frozen  as  specified  in  Table  6.1.1.  Dollar  savings  can  only  be  realized  if 
manufacturers  begin  to  build  systems  which  meet  all  these  ECDIS  standards. 


Table  6.1.1  Status  of  IMP  Compliant  ECDIS  Standards 


IMO  COMPLIANT  ECDIS  STANDARDS 

DATE 

IMO  Performance  Standard  for  ECDIS 

Nov  1995 

IHO  S57  IHO  Transfer  Standard  for  Digital  Hydrographic  Data  (Edition  3)  (note:  to 

be  held  frozen  for  four  years) 

Nov  1996 

IHO  S-52  Specifications  for  Chart  Content  and  Display  Aspects 
of  ECDIS  (Edition  5) 

Appendix  1  Guidance  on  Updating  the  ENC  (Edition  3) 

Appendix  2  Colours  and  Symbols  Specifications  for  ECDIS  (Edition  4) 

Appendix  3  Glossary  of  ECDIS  Terms  (Edition  3) 

(note:  all  of  the  above  to  be  held  frozen  for  four  years) 

Dec  1996 

Jan  1997 

Mar  1997 

Jun  1997 

lEC  Operational  and  Performance  Requirements  Methods  of  Testing  and  Required  Test 
Results  for  ECDIS  (10th  Draft  Nov  1996) 

Final 

Fall  1997 

With  ECDIS  standards  complete,  maritime  administrations  should  begin  the  process  of  testing, 
certifying,  and  type  approving  these  systems.  Requirements  and  procedures  for  training  and 
certifying  mariners  in  the  use  of  these  systems  should  also  begin  quickly.  Hundreds  of 
systems,  claiming  to  be  nearly  IMO  compliant  ECDISs  are  out  onboard  vessels  now  (e.g. 
including  the  USCG  Juniper  Class  Buoy  Tender).  Regulations  and  carriage  requirements 
should  be  established  to  insure  the  proper  and  appropriate  use  of  these  systems.  A  national 
infrastructure  to  support  ECDIS  is  needed.  One  key  element  to  this  infrastructure  is  the 
availability  of  IHO  compliant  data  and  the  mechanism  to  provide  electronic  chart  updates  to  the 
users.  It  is  only  with  a  proper  infrastructure  to  support  ECDIS  can  maritime  administrations  take 
full  advantage  of  the  safety  and  economic  benefits  ECDIS  can  bring  to  their  coimtries. 
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